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ABSTRACT 

From  October  28th  to  November  2nd  2001,  DREV  conducted  a  joint  R&D  and  military  experiment  on 
the  effectiveness  of  the  new  Lav-Recce  Enhanced  Surveillance  Demonstrator  (LRESD).  The  aim  of  this 
experiment  was  to  assess  whether  Situation  Awareness  (SA)  at  the  Command  Post  (CP)  level  was  improved  by 
adding  a  new  suite  of  sensors  on  the  standard  Coyote  vehicle,  which  constitute  the  enhanced  version,  namely, 
the  LRESD.  The  information  products  produced  by  both  Lav-Recce  suites  and  forwarded  to  the  CP  were 
compared  to  the  ground  truth  that  was  carefully  designed  prior  the  experiment.  The  timeliness  of  these 
information  products  is  also  considered.  SA  being  of  the  utmost  importance  to  the  commander  for  his  C2 
duties,  it  was  obvious  that  this  aspect  of  C2  was  to  become  our  prime  Measure  of  Effectiveness  (MoE). 
Measures  of  Performance  (MoPs)  that  would  link  both  the  systems  level  and  the  higher  MoE  were  then  chosen 
and  strategies  to  evaluate  them  identified.  This  paper  describes  the  experiment  and  how  the  experimental 
protocol  was  designed  according  to  the  principles  and  guidelines  of  the  NATO  Code  of  Best 
Practices  (COBP)  for  C2  assessment.  It  will  also  clearly  demonstrate  the  Measures  of  Effectiveness  (MoEs) 
and  Measures  of  Performance  (MoPs)  that  were  identified  to  be  the  key  measures  in  the  context  of  the 
experiment,  and  the  considerations  over  human  factors.  Finally,  the  paper  will  describe  the  lessons  that  were 
learned  during  the  experiment  that  were  not  necessarily  controlled  or  expected  with  the  positive  and  negative 
aspects  that  arose  from  them. 

Key  Words:  Experimentation,  Situation  Awareness,  Measures  of  Merit,  Information  Systems. 


1.0  INTRODUCTION 

Since  1997,  numerous  research  projects  have  been  conducted  to  improve  the  actual  sensor  suite  of  the 
Canadian  surveillance  vehicle  called  COYOTE.  The  enhanced  version  is  called  the  Eav- Recce  Enhanced 
Surveillance  Demonstrator  (ERESD).  A  secondary  goal  to  these  projects  was  to  validate  and  test  emerging 
technologies  that  exploit  and  help  disseminate  the  information  generated  by  the  vehicle  suite.  The  Defence 
R&D  Canada  -  Valcartier  (DRDC  Valcartier)  held  an  experiment  in  November  2001  that  aimed  at  evaluating 
technological  improvements  to  the  Coyote  (ERESD)  and  weigh  the  capacity  to  improve  Command  Post  (CP) 
situation  awareness  when  compared  to  a  standard  COYOTE  suite. 

This  document  describes  the  design  of  the  experiment  and  some  of  the  preliminary  results.  The  NATO  Code 
of  Best  Practice  (COBP)  [COBP,  1999]  has  been  used  as  the  general  framework  for  our  work.  The  first 

Paper  presented  at  the  RTO  SAS  Symposium  on  “Analysis  of  the  Military  Effectiveness  of  Future  C2  Concepts 
and  Systems”,  held  at  NCSA,  The  Hague,  The  Netherlands,  23-25  April  2002,  and  published  in  RTO-MP-1 17. 


RTO-MP-117 


A1  - 1 


Report  Documentation  Page 

Form  Approved 

0MB  No.  0704-0188 

Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  0MB  control  number. 

1 .  REPORT  DATE  2.  REPORT  TYPE 

00  DEC  2003  N/A 

3.  DATES  COVERED 

4.  TITLE  AND  SUBTITLE 

C2  Analysis  of  the  Effectiveness  of  the  Coyote  LRESD 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Defence  R&D  Canada  Valcartier  2459,  Pie  XI  north  Val-Belair,  Quebec 
G3J  1X5  CANADA 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release,  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

See  also  ADM001657.,  The  original  document  contains  color  images. 

14.  ABSTRACT 

15.  SUBJECT  TERMS 

16.  SECURITY  CLASSIFICATION  OF:  17.  LIMITATION  OF 

18.  NUMBER  19a.  NAME  OF 

a.  REPORT  b.  ABSTRACT  c.  THIS  PAGE 

unclassified  unclassified  unclassified 

10 

standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


C2  Analysis  of  the  Effectiveness  of  the  Coyote  LRESD 


ORGA^nZATTON 


section  of  the  document  describes  the  theoretical  background  that  supports  solid  and  objective 
experimentation.  The  second  portion  deals  with  the  design  of  the  experiment,  which  is  crucial  for  the 
identification  of  strong  and  weak  aspects  of  the  new  suite.  Also,  the  design  of  experiments  is  a  process  in 
which  there  is  much  to  learn.  As  we  will  see,  the  experimental  protocol  was  designed  to  reflect  the  state  of  the 
art  in  applying  theoretical  models  of  situation  awareness.  Whenever  possible,  quantitative  and  easy  to  measure 
metrics  (e.g.  system  metrics)  were  preferred  over  qualitative  and  hard  to  interpret  ones  (e.g.,  observer  notes). 
In  the  context  of  the  experiment,  measures  that  were  taken  in  real  time  had  higher  priority  than  the  ones  that 
needed  to  be  interpreted  after  the  exercise.  During  the  experiment,  the  emphasis  was  put  at  the  vehicle/sensor 
level  to  better  track  and  evaluate  the  passage  of  raw  reconnaissance  information  up  to  its  next  hierarchical 
command  level  for  further  processing  and  analysis  if  required.  It  was  hoped  that  by  providing  more  and 
improved  tools  to  pre-process  and  structure  reconnaissance  information  directly  at  the  vehicle  level  would 
improve  the  development  of  situation  awareness  at  the  CP  level,  therefore  reducing  redundancy  of  tasks  and 
optimizing  time  for  analysis  and  integration  into  the  overall  intelligence  picture. 


2.0  BACKGROUND  THEORY 

The  goal  of  the  LRESD-ISTAR  experiment,  as  it  is  named,  was  to  demonstrate  that  the  new  sensor  suite  better 
supports  the  operator  in  his  task  of  generating  information  products  for  higher  echelons  in  the  chain  of 
command,  in  the  context  of  the  Canadian  ISTAR  doctrine.  Two  quantities  describe  the  information  analyst’s 
performance  in  a  precise  and  complete  way:  The  accuracy  of  the  information  he  generates  for  higher  echelons, 
(the  Command  Post)  and  the  timeliness  of  his  reports.  Figure  1  illustrates  how  the  analyst’s  performance  is 
affected  by  information  accuracy  and  timeliness.  Clearly,  region  I  is  the  one  of  high  performance  while 
regions  II,  III  and  IV  are  the  ones  where  information  products  are  inaccurate,  untimely  or  worse,  both. 


Information 


Information 

Timeliness 


Figure  1:  Analyst’s  Performance. 

The  experiment  should  demonstrate  how  analyst’s  performance  is  affected  by  the  addition  of  the  new  sensors. 
Presumably,  the  performance  should  be  improved,  meaning  that  a  point  in  Figure  1  would  move  in  the  general 
northeast  direction.  It  is  therefore  important  to  understand  in  our  context  what  is  information  accuracy  and 
timeliness. 
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Situation  Awareness  as  an  MoE 

All  other  things  being  equal,  situation  awareness  is  linked  to  the  information  produet  quality.  While  situation 
awareness  is  intimately  linked  to  what  our  senses  tell  us,  it  is  reasonable  to  think  that  sensors  that  extend  our 
own  eapaeities  (hearing,  viewing,  touching,  etc.)  will  help  in  increasing  situation  awareness  and  therefore 
increasing  the  information  product  quality.  This  reasoning  constituted  the  main  driver  for  LRESD-ISTAR 
experiment.  Since,  situation  awareness  as  our  prime  Measure  of  Effectiveness  (MoE)  is  the  key  element  to 
determine  the  information  product  quality,  it  is  important  to  well  understand  how  it  is  defined  and  how  it  is 
influenced.  According  to  Endsley  [Endsley,  1988],  situation  awareness  is  the  perception  of  the  elements  in  the 
environment,  within  a  volume  of  time  and  space,  the  comprehension  of  their  meaning,  and  the  projection  of 
their  status  in  the  near  future.  Figure  2  illustrates  this  definition. 


Figure  2:  Situation  Awareness. 


In  the  context  of  the  ERESD-ISTAR  experiment,  we  were  interested  in  determining  whether  bringing  the  new 
sensor  suite  would  increase  or  not  the  operator’s  situation  awareness.  We  argued  that  assessing  level  1  of 
Endsley’s  situation  awareness  model  would  suffice  for  the  task.  We  did  not  concentrate  ourselves  on  level  2 
and  level  3,  since  the  introduction  of  new  sensors  would  not  influence  them.  Furthermore,  Jones  and  Endsley 
[Jones  and  Endsley,  1996]  report  that  76%  of  errors  attributed  to  situation  awareness  of  fighter  pilots  come 
from  problems  at  level  1  of  the  model.  This  emphasizes  the  importance  of  perception  in  Endsley’s  model.  It  is 
important  to  note  also  that  the  model  considers  temporal  aspects  like  elements  that  influence  situation 
awareness.  However,  these  aspects  have  their  influence  mostly  on  level  2  and  3  of  the  model  and  therefore  are 
of  less  relevance  to  our  case.  Of  course,  temporal  aspects  were  of  prime  importance  to  our  experiment,  but  not 
in  the  sense  Endsley  defined  them.  We  capture  temporal  aspects  in  the  concept  of  timeliness  as  discussed 
above  and  in  the  next  section. 

Timeliness  of  Information 

All  other  things  being  equal,  the  analyst’s  performance  depends  on  the  timeliness  of  his  reports.  Figure  3 
demonstrates  the  concept  of  timeliness  and  3  particular  cases.  Figure  3(a)  shows  the  general  case  where 
information  is  useless  before  ti,  useful  for  a  T2  -  ti  period,  and  finally  too  old  after  ta  (e.g.,  daily  reports  on 
refugees’  situation).  Figure  3(b)  shows  the  case  where  information  is  highly  pertinent  but  for  a  very  short 
period  of  time  (e.g.,  imminent  bombing  raid)  and  Figure  3(c)  shows  the  case  where  information  stays  pertinent 
at  all  times  (e.g.,  casualty  reports).  The  analyst’s  performance  is  linked  to  his  capacity  of  generating  and 
forwarding  information  that  is  timely,  meaning  that  his  reports  always  fall  under  the  timeliness  curve  of 
Figure  3.  Whether  this  is  an  easy  or  difficult  task  depends  on  the  shape  of  the  curve. 
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Timeliness 
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Time 


Figure  3:  Examples  of  Information  Timeliness. 


While  the  eoncept  of  timeliness  is  easy  to  grasp,  choosing  metrics  that  will  measure  it  adequately  is  rather 
difficult. 


3.0  LINKING  OUR  MoE  WITH  MoPs 

The  NATO  Code  of  Best  Practice  states  that  linking  high-level  metrics  to  system-level  ones  (MoEs-to-MoPs) 
is  very  challenging.  We  fully  concur  with  this  point.  However,  it  is  the  only  way  by  which  one  can  achieve 
valuable  C2  assessment.  This  section  deals  with  the  identification  of  2  Measures  of  Performance  (MoPs) 
and  their  linking  to  our  prime  MoE. 

All  measures  in  the  context  of  the  ERESD-ISTAR  experiment  must  be  taken  with  respect  to  the  evaluation 
objectives  that  were  stated  in  section  2.  This  also  is  consistent  with  the  NATO  COBP  guidelines.  It  is 
therefore  of  prime  importance  to  assess  the  analyst’s  accuracy  of  perception  since  it  is  a  determining  aspect  of 
the  information  product’s  quality.  The  timeliness  of  this  product  has  to  be  determined  as  the  second  parameter 
to  the  information  product’s  quality. 

In  order  to  evaluate  the  analyst’s  quality  of  perception  and  the  timeliness  of  the  information  product, 
we  literally  have  to  build  the  right  instruments  to  measure  them.  We  also  have  to  choose  carefully  where  these 
probes  will  be  placed  in  the  system.  We  need  therefore  to  understand  the  nature  of  the  observables  which  are 
the  standard  Coyote  and  the  ERESD  Coyote. 
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Description  of  the  Coyote  Vehicles 

In  essence,  the  Coyotes  are  surveillance  vehicles  equipped  with  a  certain  number  of  sensors  that  extend  the 
perception’s  capacity  of  a  human.  Table  1  enumerates  the  sensors  for  both  vehicles. 


Table  1:  Coyotes’  Sensor  Suites 


Standard  Coyote 

LRESD 

•  Visible  spectrum  camera 

•  Passive  infrared  camera 

•  Radar 

•  Visible  spectrum  camera 

•  Passive  infrared  camera 

•  Radar 

•  Active  infrared  camera 

•  Acoustic  sensor  array 

•  Coyote  Battle  Management  System  (CBMS) 

In  addition  to  its  extended  suite  of  sensors,  the  LRESD  has  a  rudimentary  information  system  that  helps  fusing 
information  that  comes  from  the  sensors.  This  system  is  called  “CBMS”  (Coyote  Battlefield  Management 
System).  It  consists  of  a  screen  that  centralizes  sensor  information,  so  it  allows  the  analyst  to  focus  on  one 
screen  instead  of  many.  CBMS  provides  tools  to  manipulate  and  adjust  the  sensors  of  the  LRESD  and  display 
their  information  on  a  map-based  interface.  It  also  allows  the  refinement  and  description  of  reconnaissance 
information  items.  Attachments  such  as  video,  annotated  photos/imagery  or  even  text  documents  can  also  be 
linked  to  the  information  items,  thus  enriching  its  “situation  awareness  value”  for  the  next  analyst. 

Both  Coyotes  have  means  to  disseminate  information  to  upper  levels  in  the  chain  of  command  (CP).  Table  2 
resumes  these  capacities. 


Table  2:  Coyotes’  Information  Dissemination  Facilities 


Standard  Coyote 

LRESD 

•  Paper  reports  (preformatted) 

•  Voice  radio  link 

•  Video  recording 

•  Structured  messaging  via  TCCCS-IRIS 

•  Paper  reports  (preformatted) 

•  Voice  radio  link 

•  Video  recording 

•  Structured  messaging  via  TCCCS-IRIS 

•  High-bandwidth  uplink  to  the  CP  via  NTDR 
radios  (full  TCP/IP). 
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Communication  links  from  the  Coyotes  to  their  respective  CPs  are  shown  in  Figure  4.  Both  vehicles 
can  transmit  information  by  voice  and  data  through  the  digital  radio  system  “TCCCS-IRIS”.  Flowever, 
transmitting  data  through  TCCCS-IRIS  requires  the  preparation  of  formatted  messages  in  USMTF  format 
mainly.  We  recall  that  in  the  context  of  this  experiment,  we  do  not  want  to  focus  on  the  treatment  of 
information  but  rather  on  its  detection  (perception).  For  this  reason,  we  denied  the  use  of  data  transmission 
through  TCCCS-IRIS,  leaving  the  standard  Coyote  with  voice  transmission  only.  The  LRESD  is  equipped 
with  an  NTDR  radio  that  basically  gives  full  TCP/IP  capabilities  over  a  287  kb/s  channel.  At  this  rate,  it  is 
reasonable  to  transmit  short  videos  and  particularly  overlays  to  the  All-Source  Intelligence  Producer  (ASIP). 
ASIP  is  a  command  and  control  information  workbench  prototype  that  supports  the  intelligence  operator  in  his 
task.  Basically,  the  information  produced  by  the  LRESD  was  sent  over  the  NTDR  link  and  reproduced  on  an 
ASIP  overlay  in  the  CP. 


TCCCS  (Voice) 


NTDR  (Data) 


LRESD 


Figure  4:  Communications  Links. 


Although  the  communications  links  of  the  2  configurations  are  different  (voice  vs.  data)  and  that  the  receiving 
ends  (ASIP  vs.  CP  analyst)  are  different  also,  there  is  no  impact  on  our  experiment  because  we  chose  to  record 
the  state  of  the  information  at  the  moment  where  the  information  products  quit  the  vehicles  and  not  the 
moment  they  arrive  at  the  CP  (or  later).  In  fact,  the  use  of  ASIP  fulfilled  other  goals  that  were  of  no  relevance 
to  this  experiment.  On  the  other  hand,  the  information  products  quality  was  determined  by  a  judge  afterwards. 

Coyotes  Deployment 

In  order  to  verify  the  impact  the  new  sensors  and  CBMS  have  on  the  analyst’s  situation  awareness,  it  is 
important  that  all  other  parameters  that  might  influence  it  be  kept  constant.  One  of  these  parameters  is  the 
position  of  the  vehicles.  Both  vehicles  were  given  the  same  zone  of  surveillance.  Within  this  zone,  events  that 
would  elicit  responses  from  different  sensors  occurred.  Although  the  vehicles  were  apart  (about  100  feet), 
their  sensor  suites  were  collocated,  giving  equal  chances  of  detection.  Of  course,  the  nature  of  certain  events 
would  not  trigger  equally  both  sensor  suites,  and  that’s  the  point  of  making  the  experiment.  This  is  the 
case  notably  for  acoustic  events,  which  are  only  detectable  through  the  acoustic  array  of  the  LRESD. 
Sensor  detection  does  not  necessarily  mean  human  detection  and  it  might  occur  that  certain  acoustic  events 
would  not  be  detected  (and  reported)  by  the  intelligence  analyst.  The  point  was  to  find  why.  Of  all  the  hard 
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data  that  we  obtained  from  this  experiment,  notes  from  passive  observers  were  probably  the  most  important 
source  of  discovery  for  improving  our  systems.  We  discovered  hidden  aspects  of  the  system  that 
had  significant  impact  on  the  way  information  flow  was  influenced.  Some  of  these  aspects  were  good 
(e.g.,  inferring  information  from  the  toggling  between  2  sensors),  others  were  bad  (e.g.,  overloading  the 
analyst’s  job). 


4.0  SYSTEM-LEVEL  METRICS  (MoPS) 

The  Coyotes  and  their  information  analysts  constitute  systems  that  accept  inputs  (sensor  information)  and  give 
outputs  (information  product).  In  a  simplified  way,  an  output  y(t)  is  a  representation  of  the  input  with  a  certain 
scaling  factor,  and  delayed  by  a  certain  period  of  time.  So, 

y(t)  =  a-x(t  -  t) 


Distortion  Factor  a 

The  scaling  factor  a  represents  the  situation’s  quality  of  perception  of  the  system  (sensor  suite  +  intelligence 
analyst).  This  distortion,  which  may  be  non-linear,  depends  on  the  quality  of  the  sensor-analyst  set.  By  setting 
the  analyst’s  competence  to  a  constant,  the  distortion  factor  a  depends  on  the  sensor  suite  quality.  With  a 
series  of  identical  events  presented  to  both  Coyote  vehicles,  we  obtained  a  direct  measure  of  the  quality  of 
perception  and  therefore  a  measure  of  the  gain  in  situation  awareness.  This  measure  depended  on  the  sensor 
suites.  In  practice,  this  factor  is  evaluated  in  the  CP  by  a  judge  as  a  function  of  the  difference  between  the 
information  reported  and  the  ground  truth.  The  distortion  factor  is  evaluated  along  these  guidelines: 

•  Object  detected  or  not 

•  Correct  identification  of  the  object 

•  Accuracy  of  the  reported  position  of  the  object 

•  Estimated  object  speed 

•  Assessed  activity  of  the  object 

•  Projected  behavior  of  the  object 

Delay  Factor  t 

The  delay  factor  T  is  defined  as  the  elapsed  time  between  the  moment  where  an  event  occurs  and  the  moment 
where  this  event  is  actually  reported  to  the  CP  by  the  information  analyst.  It  is  equal  to  the  sum  of  the  delay  of 

sensor  detection  (Zj;),  the  delay  of  analyst  detection  (Zh),  and  the  delay  of  analyst  information  processing  (Zj,). 
So, 

T  Tp 

We  supposed  that  information  would  not  be  intentionally  retained  by  analyst  for  over-processing. 
Also,  we  judged  that  sensor  detection  delays  would  be  negligible  when  compared  to  the  analyst  detection  and 

information  processing  delay.  T  is  influenced  by  several  factors  like  the  analyst’s  attention,  sensors 
ergonomics,  information  ambiguities,  information  overloading,  etc.  It  is  important  to  understand  that  this 
delay  is  normal  and  that  it  only  gains  significance  when  compared  to  the  timeliness  profile  of  this 
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event.  For  example,  a  delay  T situated  between  T\  and  T2  in  Figure  3(a)  is  perfeetly  acceptable,  while  a  delay 

T  longer  T\  than  in  Figure  3(b)  is  alarming.  Timeliness  profiles  for  each  event  (or  group  of  events)  were  to  be 
known  prior  the  experiment  in  order  to  evaluate  the  analysts’  performance  against  temporal  aspects. 

Practical  Considerations 

While  a  and  T  were  measurable  by  introducing  probes  at  the  right  places  in  the  system,  we  preferred  using 
passive  educated  observers  as  the  main  measuring  instruments  for  this  experiment.  “Passive”  observers  are 
non-intrusive  agents.  They  do  not  interfere  with  the  analyst’s  job,  they  do  not  help  nor  do  they  comment  the 
analyst’s  behaviour.  This  is  important  since  it  would  introduce  significant  distortion  in  both  temporal  and 
quality  metrics.  “Educated”  observers  know  what  events  will  occur,  when  they  will  occur.  This  prior 
knowledge  enables  them  to  focus  their  observations  to  what  is  important  to  note  and  therefore  explain  why 
certain  delays  are  particularly  long  (e.g.  an  analyst  distracted  by  something). 

Someone  might  say  that  using  humans  as  observers  may  introduce  subjectivity  to  the  outcome  of  the 
experiment.  We  argue  that  certain  aspects  of  systems  (including  human-in-the-loop)  are  simply  not  possible  to 
evaluate  without  observers.  Two  aspects  of  using  observers  must  be  avoided  at  all  costs.  The  first  is  an 
observer  with  unclear  objectives.  Fie  will  observe  things  all  right,  but  without  any  focus  on  the  assessment 
objectives,  the  observations  become  useless.  The  second  one  is  an  observer  in  a  mechanized  role.  By  knowing 
the  nature  of  the  measures  he  takes,  an  observer  can  comment  aspects  of  his  measures,  enabling  statistical 
analysis  after  the  experiment.  If  not,  numbers  will  be  numbers  and  some  won’t  be  explainable. 


5.0  SUMMARY 

While  the  data  obtained  in  the  three-day  experiment  is  still  being  analyzed  and  that  we  cannot  present  fancy 
graphics  yet,  we  still  can  talk  about  the  lessons  learned.  First,  we  noted  that  the  degree  of  integration  of 
CBMS,  the  LRESD  information  system,  was  not  sufficient  to  support  efficiently  the  analyst’s  task  in  the 
vehicle.  Indeed,  the  analyst  had  to  jump  from  a  sensor  to  another  in  order  to  acquire  enough  information  and 
this  lead  to  increased  delays  in  the  processing.  Second,  we  noticed  that  sensor  sweeping  counted  for  a  fair 
portion  in  the  delay  of  detection,  especially  at  nights.  Fortunately,  we  were  able  to  evaluate  these  delays 
enough  for  compensation.  Third,  the  acoustic  array  is  a  sensor  suite  that  triggers  the  interest  of  the  analyst. 
At  best,  it  gives  the  bearing  and  range  of  a  certain  target.  The  analyst  must  then  rely  on  the  other  sensors  to 
effectively  confirm  and  identify  the  target.  While  this  new  sensor  may  help  greatly  the  analyst  by  focusing  his 
attention,  the  benefit  of  having  an  acoustic  array  may  not  have  been  measured  with  the  metrics  we  chose. 
In  fact,  we  feel  that  we  did  not  measure  it  well.  Empirically,  our  results  show  no  benefit  of  having  this  type  of 
sensor,  but  this  is  not  because  there  is  no  benefit  of  having  it.  It  was  merely  a  question  of  choosing  the  right 
metrics.  Fourth,  there  are  events  that  are  intrinsically  hard  to  detect.  Snipers  on  observation  mission  are  one 
example.  In  this  case,  a  reconnaissance  vehicle  is  almost  lucky  to  detect  one  even  with  the  LRESD  extended 
sensor  suite.  This  inserted  great  distortions  in  our  results.  Fifth,  giving  new  tools  to  the  analyst  has  a 
non-negligible  impact  on  the  standard  operating  procedures  (SOPs).  New  sensors  mean  new  ways  of 
interpreting  information  and  therefore  new  ways  of  doing  things.  The  analyst’s  job  is  impacted  and  parameters 
that  should  have  stayed  constant  throughout  the  experiment  might  have  varied  without  us  knowing. 

These  are  but  a  few  things  that  we  learned  in  this  experiment.  The  application  of  the  experiment  protocol  was 
a  complex  task  given  the  many  parameters  and  uncertainties  typical  to  military  operations.  The  NATO  COBP 
helped  us  designing  the  protocol,  particularly  concerning  the  choice  of  metrics.  Conducting  experiments  is  a 
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process  that  one  can  only  learn  through  experience.  It  left  us  with  many  valuable  lessons.  We  hope  to  have 
transmitted  some  of  these  to  the  reader. 
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7.0  LIST  OF  ACRONYMS 

ASIP  All-Source  Intelligence  Producer 

C2  Command  and  Control 

CBMS  Coyote  Battlefield  Management  System 

COBP  Code  of  Best  Practice 

CP  Command  Post 

DREV  Defence  Research  Establishment  Valcartier 

ISTAR  Intelligence  Surveillance  Target  Acquisition  and  Reconnaissance 

ERESD  Eav-Recce  Enhanced  Surveillance  Demonstrator 

MoE  Measure  of  Effectiveness 

MoP  Measure  of  Performance 
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